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DETAILED ACTION 

This office action is responsive to communication filed February 1 1 , 2008. 
Claims 1 0, 1 1 , 1 5, 1 7, 1 8, 21 , 23 and 24 are currently amended, claim 1 6 is additionally 
canceled, and claims 27 and 28 have been added. Therefore claims 10, 11, 15, 17, 18 
and 20-28 are pending in this application. 



Claim Rejections - 35 USC § 102 

1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351 (a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

2. Claim 10, 20-23 are rejected under 35 U.S.C. 102(e) {current application filing 
date 1/26/2004) as being anticipated by Dawe et al (US 7,042,594 B1, filing date 



3/7/2000) ('Dawe'). 
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With respect to claim 10, Dawe teaches: 

receiving a user input (col. 6 lines 48-50) defining boundaries of a selected 
region of a display, one or more graphical elements being displayed in the selected on- 
screen region (col. 6 lines 42-51); 

capturing image pixels for displaying the one or more graphical elements, and 
storing the captured image pixels in an image file (col. 3 lines 60-63, col. 7 lines 40-42 
and lines 61-65, col. 8 lines 1-3); 

obtaining context information for the one or more graphical elements by 
automatically performing at least one of the following: 

applying text recognition to an annotation drawn by the user on the display in 
proximity of the selected on-screen region, and storing the results of the text recognition 
as context information; 

determining whether the one or more graphical elements represent textual data, 
extracting a character or word from the textual data, and storing the extracted character 
or word as context information (col. 5 lines 36-43, col. 7 lines 57-60); and 

determining whether the one or more graphical elements is associated with 
underlying data, extracting a property of the underlying data from an application causing 
the one or more graphical elements to be displayed when executed in the computer- 
based system, and storing the extracted property as context information, 

wherein the context information is automatically stored in association with the 
image file (col. 7 lines 57-60). 
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With respect to claim 20, Dawe teaches wherein the context information is stored 
in such a manner as to be accessible to a user for performing at least one of the 
following: 

searching for said image file; 

displaying the context information simultaneously with the captured image pixels 
(col .8 lines 1-16), and 

navigating a network to a source of the captured image pixels. 

With respect to claim 21 , Dawe teaches wherein the one or more graphical 
elements comprises a first set of one or more textual characters, the method further 
comprising: obtaining the context information as text data obtained by performing text 
recognition on at least one of: the first set of one or more textual characters, and a 
second set of textual characters displayed in proximity with the first set (col. 7 lines 6-20 
and lines 61-65). 

With respect to claim 22, Dawe teaches wherein the selected on-screen region is 
part of displayed textual region, and the graphical elements comprise a first set of one 
or more textual characters displayed in the textual region, the method further 
comprising: obtaining the context information based on a second set of one or more 
textual characters displayed in the textual region (Fig. 4, col. 7 lines 6-20 and lines 61- 
65) {The word "job" may be considered the second set of textual characters). 
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With respect to claim 23, Dawe teaches wherein the step a) receives the user 
input based on movement of a stylus across the display (col. 5 lines 28-30). 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claim 1 1 , 25 and 26 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Dawe et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Oppermann 
et al. (US 6,334,157 B1 , filing date 3/11/1997) ('Oppermann'). 

With respect to claim 1 1 , Dawe teaches claim 10, including a selected on-screen 

region. 

Dawe does not teach determining a window associated with the selected on- 
screen region; retrieving an application interface having a uniform resource identifier 
(URI) property from the determined window or parent window of the determined 
window; or obtaining the URI property as the context information. 

Oppermann teaches programmatically providing direct access to user interface 
elements of an application program (see abstract), in which he teaches: 

selecting user interface elements, such as text (col. 8 lines 43 and 49-51) and 
determining a window associated with the selected elements (col. 26 lines 37-40); 
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retrieving an application interface having a uniform resource identifier (URI) 
property from the determined window or parent window of the determined window (col. 
25 lines 59-62, col. 28 lines 33-39); and 

obtaining the URI property as the context information (col. 1 1 lines 1-9, 55-60, 
col. 12 lines 55-60, col. 13 lines 51-60). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Dawe by the teaching of Oppermann because 
determining a window, retrieving an application interface having a uniform resource 
identifier (URI) property from the determined window or parent window of the 
determined window would enable accessibility aids the ability to access and manipulate 
user interface elements of any application program without having prior knowledge of 
the application program or its interface (Oppermann, column 4 lines 27-30). 

With respect to claim 25, Dawe as modified teaches wherein the selected on- 
screen region includes at least a portion of a displayed web page or document (Dawe, 
Figure 4, col. 4 lines 38-42), and the step d) further comprises: using an application 
programming interface (API) to query the application for the context information 
(Oppermann, column 7 lines 36-38). 

With respect to claim 26, Dawe as modified teaches wherein the step d) further 
comprises obtaining a uniform resource identifier (URI) of the web page or document as 
the context information (Oppermann, col. 11 lines 1-9, 55-60, col. 12 lines 55-60, col. 13 
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lines 51 -60), the URI being obtained as a result of the query using the API (Oppermann, 
column 7 lines 36-38). 

5. Claims 15 and 18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Dawe et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Browne et 
al. (US 2004/0135815 A1, filing date 12/15/2003) ('Browne'). 

With respect to claim 15, Dawe teaches claim 10. 

Dawe does not teach creating and storing a linking structure as the association 
between the image file and the context information. 

Browne teaches a method and apparatus for image metadata entry (see 
abstract), in which he teaches creating and storing a linking structure as the association 
between the image file and the context information (Figure 12, paragraph 136). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Dawe by the teaching of Browne because 
teach creating and storing a linking structure as the association between the image file 
and the context information would enable an easy and efficient method of classifying 
and storing digital images of the scanned documents of Dawe (Browne, paragraph 14). 

With respect to claim 18, Dawe as modified teaches wherein the linking structure 
includes at least one pointer pointing to the stored image file or the stored content 
information (Browne, paragraph 136). 
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6. Claim 17 is rejected under 35 U.S.C. 103(a) as being unpatentable over Dawe et 
al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Browne et al. (US 
2004/0135815 A1 , filing date 12/15/2003) ('Browne') as applied to claims 15 and 18 
above, and further in view of Newman (US 2003/0101 156 A1, filing date 11/26/2001). 

With respect to claim 17, Dawe as modified teaches claim 15. 

Dawe as modified does not teach wherein the linking structure is incorporated in 
a file separate from the stored image file and the stored content information. 

Newman teaches database systems and methods (see abstract), in which he 
teaches wherein the linking structure is incorporated in a file separate from the stored 
image file and the stored content information (paragraph 16). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Dawe by the teaching of Newman 
because wherein the linking structure is incorporated in a file separate from the stored 
image file and the stored content information would enable additional information about 
image files, such as the origination device, person who created the file, and data/time 
the file was created, to be transmitted and stored along with the image files (Newman, 
paragraph 16). 
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7. Claims 24 is rejected under 35 U.S.C. 103(a) as being unpatentable over Dawe 
et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Saund et al. (US 
2003/01 82630 A1 , filing date 3/22/2002) ('Saund'). 

With respect to claim 24, Dawe teaches claim 10. 

Dawe does not teach digitizing movements of a stylus across the display in order 
to receive the user annotation; or obtaining the context information based on the 
received annotation. 

Saund teaches a method for gestural interpretation in a system for selecting an 
arranging visible material in document images (see abstract), in which he teaches: 

digitizing movements of a stylus across the display in order to receive the user 
annotation (Figure 3, paragraph 26); and 

obtaining the context information based on the received user annotation 
(paragraph 31). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Dawe by the teaching of Saund 
because digitizing movements of a stylus across the display in order to receive the user 
annotation; or obtaining the context information based on the received annotation would 
enable the ability to execute unimpeded sequential input forfreeform strokes of many 
types rapidly and without making explicit choices about user interface modes (Saund, 
paragraph 34). 
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8. Claims 27 is rejected under 35 U.S.C. 103(a) as being unpatentable over Dawe 
et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Hertzfeld et al. (US 
2002/0076109 A1 , filing date 1/25/1999) ('Hertzfeld') 

With respect to claim 27, Dawe teaches: 

receiving a path drawn on the display by a user via a stylus (col. 5 lines 28-30; 
col. 6 lines 48-50), the drawn path defining the boundaries of a selected on-screen 
region of the display (col. 6 lines 42-51); 

capturing each pixel within the boundaries of the on-screen region (col. 3 lines 
60-63, col. 7 lines 40-42 and lines 61-65, col. 8 lines 1-3); 

storing the captured pixels as an image file (col. 3 lines 60-63, col. 7 lines 40-42 
and lines 61-65, col. 8 lines 1-3); 

automatically determining whether the content displayed within the on-screen 
region includes textual data (col. 6 lines 52-53, col. 7 lines 11-14); 

if the displayed content of the on-screen region is determined to include textual 
data, automatically extracting a character or word from the textual data as context 
information (col. 7 lines 6-8 and 15-20); and 

storing the context information in association with the image file, such that the 
context information is accessible when viewing the image file (col. 7 lines 57-60, col. 8 
lines 1-16). 

Dawe does not teach automatically determining whether the displayed content of 
the on-screen region includes underlying data comprising at least one of: an 
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executable object, a file, and a link to remote content; or if the displayed content of the 
on-screen region is determined to include underlying data, automatically extracting a 
property of the underlying data as context information, the property comprising at least 
one of: a file name, a file identifier, a uniform resource locator (URL), a uniform 
resource identifier (URI), a folder name, and meta-data. 

Hertzfeld teaches a method and apparatus for context sensitive text recognition 
(see abstract), in which he teaches: 

automatically determining whether the displayed content of the on-screen region 
includes underlying data comprising at least one of: an executable object, a file, and a 
link to remote content (step 608 in Figure 6, paragraph 38) 

if the displayed content of the on-screen region is determined to include 
underlying data, automatically extracting a property of the underlying data as context 
information, the property comprising at least one of: a file name, a file identifier, a 
uniform resource locator (URL), a uniform resource identifier (URI), a folder name, and 
meta-data (step 616 in Figure 6, paragraph 38). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Dawe by the teaching of Hertzfeld because 
automatically determining whether the displayed content of the on-screen region 
includes underlying data comprising at least one of: an executable object, a file, and a 
link to remote content; or if the displayed content of the on-screen region is determined 
to include underlying data, automatically extracting a property of the underlying data as 
context information, the property comprising at least one of: a file name, a file identifier, 
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a uniform resource locator (URL), a uniform resource identifier (URI), a folder name, 
and meta-data would enable recognition of predefined types of text and predefined 
actions to be performed based on the types of text (Hertzfeld, abstract). 

The limitations "automatically extracting a character or word from the textual data 
as context information" and "automatically extracting a property of the underlying data 
as context information, the property comprising at least one of: a file name, a file 
identifier, a uniform resource locator (URL), a uniform resource identifier (URI), a folder 
name, and meta-data" are conditional statements, and thus optionally patentable. The 
limitations only occur IF another action occurs, and thus do not limit the claim. 

9. Claims 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over Dawe 
et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Beauchamp et al. (US 
2008/0046837 A1 , filing date 3/17/2003) ('Beauchamp'). 

With respect to claim 28, Dawe teaches: 

receiving a path drawn on the display by a user via a stylus (col. 5 lines 28-30; 
col. 6 lines 48-50), the drawn path defining the boundaries of a selected on-screen 
region of the display (col. 6 lines 42-51); 

capturing each pixel within the boundaries of the on-screen region (col. 3 lines 
60-63, col. 7 lines 40-42 and lines 61-65, col. 8 lines 1-3); 

storing the captured pixels as an image file (col. 3 lines 60-63, col. 7 lines 40-42 
and lines 61-65, col. 8 lines 1-3); 
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performing text recognition on an annotation to produce recognized text of the 
annotation as context information (col. 5 lines 36-43, col. 7 lines 6-20, col. 8 lines 19- 
29); 

automatically determining whether the content displayed within the on-screen 
region includes textual data or other underlying data (col. 6 lines 52-53, col. 7 lines 11- 
14); 

if the displayed content of the on-screen region is determined to include textual 
data, automatically extracting a character or word from the textual data as context 
information (col. 7 lines 6-8 and 15-20); 

if the displayed content of the on-screen region is determined to include 
underlying data, automatically extracting a property of the underlying data as context 
information (col. 8 lines 19-26); and 

storing the context information in association with the image file, such that the 
context information is accessible when viewing the image file (col. 7 lines 57-60, col. 8 
lines 1-16). 

Dawe does not teach receiving an annotation drawn on the display by the user 
via the stylus; or performing text recognition of the annotation (drawn on the display) to 
produce recognized text of the annotation as context information. 

Beauchamp teaches a transparent windows method and apparatus (see 
abstract), in which he teaches: 

receiving an annotation drawn on the display by the user via the stylus 
(paragraph 6); and 
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performing text recognition of the annotation (drawn on the display) to produce 
recognized text of the annotation as context information (paragraph 6). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Dawe by the teaching of Beauchamp 
because receiving an annotation drawn on the display by the user via the stylus; or 
performing text recognition of the annotation (drawn on the display) to produce 
recognized text of the annotation as context information would enable optimization of 
pen-based annotations on a window of a software application (Beauchamp, paragraph 
8). 

The limitations "automatically extracting a character or word from the textual data 
as context information" and "automatically extracting a property of the underlying data 
as additional context information" are conditional statements, and thus optionally 
patentable. The limitations only occur IF another action occurs, and thus do not limit 
the claim. 

Response to Arguments 

1 0. Applicant's arguments with respect to claims 10, 11, 15, 17, 18 and 20-28 have 
been considered but are moot in view of the new ground(s) of rejection. 

Conclusion 

1 1 . Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See M PEP 
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§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Alicia M. Lewis whose telephone number is 571-272- 
5599. The examiner can normally be reached on Monday - Friday, 9 - 6:30, alternate 
Friday off. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Charles Rones can be reached on 571-272-4085. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 



Application/Control Number: 10/766,319 Page 16 

Art Unit: 2164 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/Alicia M Lewis/ 
Examiner, Art Unit 2164 
May 12, 2008 

/Charles Rones/ 

Supervisory Patent Examiner, Art Unit 2164 



